Microscopic image capturing method and microscopic image capturing device

ABSTRACT

A microscopic image capturing method includes: emitting a light beam from a light beam light source; detecting a spot image by a camera or a photodiode; focusing the spot image by an objective lens actuator moving an objective lens in the optical axis direction of the light beam with respect to a sample container; determining by a reflective surface identification unit whether the light beam has been applied to a defect on the basis of the spot image; when it is determined that the light beam has been applied to the defect, moving by an XY stage the sample container with respect to the objective lens in a direction orthogonal to the optical axis of the light beam in accordance with a prescribed condition; and, when the light beam has not been applied to the defect, capturing a microscopic image of a sample by using an illuminator.

TECHNICAL FIELD

The present invention relates to a microscopic image capturing methodand a microscopic image capturing device.

BACKGROUND ART

As a method for measuring a shape of a fine particle such as a cell in aliquid, there is mainly a light scattering method for analyzingscattering of light and an image imaging method for capturing an imageof a particle with a microscope or the like.

The size and number of high-concentration particles can be easilymeasured according to the light scattering method. Therefore, the lightscattering method is generally used as a method for inspecting drugactivity against cells and bacteria. For example, a liquid containingbacteria is irradiated with light, incident light is scattered by thebacteria, and attenuation in an amount of transmitted light is measured.Accordingly, a growth state of the bacteria is measured.

As a medical device for bacteria using this principle, there is a device(sensitivity inspection device) that inspects an effect of anantimicrobial agent on bacteria, that is, an effect of inhibiting growthof bacteria by the antimicrobial agent. For example, in a sensitivityinspection device, the number of bacteria is counted using the lightscattering method.

However, the light scattering method has low sensitivity and requiresculture for one day and night. Culture for a long period of time isrequired in this way, and thus speeding up of a sensitivity inspectiondevice has been actively studied in recent years.

As a method for achieving speeding up, it is necessary to improvesensitivity. In order to improve sensitivity, a method for increasing aninitial culture concentration of bacteria is considered, but the initialconcentration of bacteria is determined to be a low concentration of 5 ×10⁵ CFU/ml by organizations such as the Clinic and Laboratory StandardsInstitute (CLSI), and the concentration of bacteria cannot be changed.

Regarding improvement in sensitivity in detection of cells and bacteria,the size and number of cells can be detected with high sensitivityaccording to the image imaging method. Since not only the size andnumber of cells but also a shape of the cell can be measured regardingto an amount of information, more information can be obtained about astate of the cells than in the light scattering method.

Known means for observing the state of cells according to the imageimaging method include optical microscope observation, fluorescencemicroscope observation, a coherent anti-Stokes Raman scattering (CARS)microscope, three-dimensional optical coherence tomography (OCT), andthe like.

In these image imaging methods, it is important to analyze an accurateshape of a cell, and thus it is essential to accurately focus on thecell. In a research stage, manual focus by human hands is often used forfocusing. In fields requiring acquisition of a huge amount of images,such as drug discovery and medical care, high-speed and high-accuracyautofocus is essential.

When a microscopic image of a cell is acquired, the cell is suspended ina liquid sample, and a container that is transparent for light to beobserved is used as a sample container that accommodates the liquidsample.

It is appropriate to dispose an objective lens below the samplecontainer. The reason is that since the cell to be observed is adheredto a boundary between the sample liquid and the sample container (forexample, an inner side of a bottom surface of the sample container), thecell is observed through the sample liquid when the sample is observedfrom above the sample container, and aberration correction becomesdifficult due to a liquid surface of the liquid being a complicatedcurved surface or the liquid surface being shaken by vibration orconvection of air.

When the sample is observed from below the sample container, aberrationcorrection can be easily performed since the sample container is solidand a thickness of the bottom surface is fixed. An inverted microscopethat observes a sample container from below is generally used for cellobservation. A focus position of the lower objective lens is in thevicinity of a position where the liquid sample is in contact with thesample container. In the liquid sample, the cell is suspended.

There are generally two types of autofocus methods for automaticallyadjusting a focus position of the objective lens, that is, an imagemethod for calculating a focus position based on contrast of amicroscopic image and an optical method for emitting light for autofocussuch as a laser and calculating a focus position based on the reflectedlight.

In autofocus according to the image method, a plurality of microscopeimages are acquired while changing a position of the objective lens, anda position where contrast is maximized is measured. In this method, inorder to complete the autofocus at high speed, a program or a device forcalculating contrast at high speed for the plurality of microscopeimages is required, and it is difficult to improve an autofocus speed inprinciple. In the microscopic images used for the autofocus, it isnecessary to use an object from which contrast can be obtained as anobservation target, and the autofocus cannot be performed with asuspension of extremely dilute cells.

On the other hand, in autofocus according to the optical method, aboundary surface between a sample container and a sample liquid isirradiated with light such as a laser, and the autofocus is performedbased on information such as a position and a phase of light reflectedfrom the boundary surface (for example, one numerical value or severalscalar quantities). Therefore, when a microscopic image having severalhundred thousand pixels is acquired, an amount of calculation in theoptical method is ⅟100,000 or less of that in the image method, andhigh-speed autofocus can be performed.

Since the autofocus is performed only by the reflected light from theboundary surface between the sample container and the sample liquid, theautofocus can be performed even in a state in which there is no objectsuch as a cell. From the object such as a cell, contrast can beobtained.

Here, regarding application of the autofocus according to the opticalmethod to the microscope, it is important that an autofocus surface ofthe sample container is a smooth portion in order to reflect lighttherefrom. Therefore, when there is a defect (such as a protrusion in apattern) in a focus surface irradiated with light such as a laser forautofocus, a focus position may be greatly deviated from the targetfocus surface.

Therefore, various methods have been devised in order to avoid a portionhaving a defect on a focus surface and to perform autofocus on a portionhaving no defect. PTLs 1 and 2 disclose examples of such a method.

For example, in a method according to PTL 1, a surface shape is acquiredand stored in advance by performing autofocus on a surface having nodefect and different from a focus surface whose image is to be actuallycaptured, and an image of the focus surface whose image is to beactually captured is captured based on data on the surface shape.

In a method according to PTL 2, four irradiation points of light such asa laser for autofocus are provided on a focus surface on which autofocusis performed, whereby one point where an irradiation point overlaps aportion having a defect is invalidated, and appropriate autofocuscontrol is performed based on the other points.

Citation List Patent Literature

-   PTL 1: JP-A-2000-294608-   PTL 2: JP-A-2001-305420

SUMMARY OF INVENTION Technical Problem

In the related art, there is room for improvement in processing forperforming autofocus on a portion having no defect.

For example, in the method according to PTL 1, throughput is low sinceit is necessary to acquire the surface shape in advance.

The method according to PTL 1 cannot be applied to a case where there isno surface having no defect.

In the method according to PTL 2, types of defects that can bedetermined are limited. For example, in a case where a defect is aprotrusion in a designed pattern or the like, it is possible todetermine that an irradiation point is invalid data based on a signalfrom a light reception unit that receives reflected light from theirradiation point, while in a case where a defect is a minute crack or aforeign substance on a focus surface, it is difficult to determine thedefect based on a signal of reflected light since a three-dimensionalshape or optical characteristics of the defect are different.

In the method according to PTL 2, throughput is low since it isnecessary to perform light irradiation and determination at four pointsin order to complete autofocus control once.

The invention has been made to solve such problems, and an objectthereof is to provide a microscopic image capturing method and amicroscopic image capturing device capable of improving processing forperforming autofocus on a portion having no defect.

Solution to Problem

An aspect of a microscopic image capturing method according to theinvention is a microscopic image capturing method for capturing amicroscopic image using a microscopic image capturing device.

The microscopic image is a microscopic image of a cell or a particle asa sample in contact with an inner side of a bottom surface of acontainer.

The microscopic image capturing device includes:

-   a transparent container configured to accommodate a sample;-   a light source for microscope imaging;-   a light beam light source configured to emit a light beam toward the    inner side of the bottom surface of the container;-   an objective lens used to form a spot image of the light beam    reflected by the sample or the container;-   a detector configured to detect the formed spot image; and-   a moving mechanism configured to relatively move the container and    the objective lens.

In the method, the microscopic image capturing device further includes adetermination unit configured to determine whether the light beam hasbeen applied to a defect based on the spot image.

The microscopic image capturing method includes:

-   a step of emitting the light beam from the light beam light source;-   a step of detecting the spot image by the detector;-   a step of focusing the spot image by the moving mechanism moving the    objective lens in an optical axis direction of the light beam with    respect to the container;-   a step of determining by the determination unit whether the light    beam has been applied to a defect based on the spot image,-   a step a) of, when it is determined that the light beam has been    applied to the defect, moving by the moving mechanism the container    with respect to the objective lens in a direction orthogonal to an    optical axis of the light beam according to a prescribed condition;    and-   a step b) of, when it is determined that the light beam has not been    applied to the defect, capturing a microscopic image of the sample    using the light source for microscope imaging.

An aspect of a microscopic image capturing device according to theinvention is configured to perform the above-described method.

Advantageous Effects of Invention

According to the microscopic image capturing method and the microscopicimage capturing device in the invention, processing for performingautofocus on a portion having no defect is improved.

For example, according to an embodiment of the invention, even on afocus surface having a defect, it is possible to maintain high-accuracyautofocus by avoiding the defect. In addition, it is possible to copewith various defects, and it is possible to cope with unexpected randomdefects in some cases.

Further features related to the invention are clarified based ondescription of the present specification and accompanying drawings. Inaddition, problems, configurations, and effects other than thosedescribed above will be clarified by description of the followingembodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view showing a configuration of a device accordingto first, second, and third embodiments of the invention.

FIG. 2 is a schematic view showing an optical system of the device inFIG. 1 .

FIG. 3 is a schematic view showing a measurement method using the devicein FIG. 1 .

FIG. 4 is a flowchart of autofocus processing of the device according tothe first embodiment.

FIG. 5 is a view showing variations of spot images.

FIG. 6 is an example of a determination criterion according to the firstembodiment.

FIG. 7 is another example of the determination criterion according tothe first embodiment.

FIG. 8 is a flowchart of processing in a learning stage of the deviceaccording to the second embodiment.

FIG. 9 shows examples of a progress image during execution of theautofocus processing.

FIG. 10 is a configuration example of teacher data according to thesecond embodiment.

FIG. 11 is a flowchart of autofocus processing of the device accordingto the second embodiment.

FIG. 12 is a flowchart of autofocus processing of the device accordingto the third embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the invention will be described withreference to the accompanying drawings. Although the accompanyingdrawings show specific embodiments according to the principle of theinvention, the accompanying drawings are shown for a purpose ofunderstanding the invention, and are not to be used for limitinginterpretation of the invention.

Each of the following embodiments relates to a microscopic imagecapturing device. The microscopic image capturing device is a deviceused to capture a microscopic image, and is, for example, an observationdevice for observing a sample. Each embodiment relates to a microscopicimage capturing method performed by the microscopic image capturingdevice.

A sample whose image is to be captured as the microscopic image is acell or a particle. Here, a size of the particle is as desired, and isnot limited to that of a fine particle. The sample is accommodated in atransparent sample container and is in contact with an inner side of abottom surface of the sample container. The sample may be contained in aliquid to form a sample liquid.

Each of the following embodiments relates to a device that automaticallyacquires a microscopic image of a cell present on a boundary surfacebetween a sample liquid and a bottom surface of a sample container inthe sample container by autofocus. Hereinafter, an outline of anobservation device common in each embodiment will be described first,and then a measurement method using the observation device in eachembodiment will be described.

An XYZ Cartesian coordinate system is set in the following description.A prescribed direction in a horizontal plane is an X direction, adirection orthogonal to the X direction in the horizontal plane is a Ydirection, and a direction orthogonal to each of the X and Y directions(that is, a vertical direction) is a Z direction. The Z directioncoincides with an optical axis of an objective lens in the followingexample.

1. Outline of Observation Device

FIG. 1 shows an overall appearance of the observation device. Theobservation device functions as the microscopic image capturing deviceand performs the microscopic image capturing method.

As shown in FIG. 1 , the observation device includes a sample container100 (container), an imaging unit 200, an XY stage 300 (movingmechanism), and a control PC 400. The XY stage 300 relatively moves thesample container 100 and the imaging unit 200 (in particular, anobjective lens 202 to be described later) in a plane orthogonal to anoptical axis of the imaging unit 200. The XY stage 300 may move thesample container 100, the imaging unit 200, or both.

The sample container 100 may be, for example, a 96-well microtiterplate, or may include a plurality of sample holders 101. A 384-wellplate, a 1536-well plate, or the like having more sample holders 101 maybe used.

The imaging unit 200 is an optical system of an inverted microscope, andincludes the objective lens 202, an objective lens actuator 203 (movingmechanism), a camera 204, an image forming lens 205, an autofocus unit206, a focus control unit 207, a reflective surface identification unit208, and an illuminator 209 (light source for microscope imaging).

The objective lens 202 is used to form a spot image of a light beamreflected by a sample or the sample container 100. The objective lens202 is also used to capture a microscopic image of the sample.

The objective lens actuator 203 relatively moves the sample container100 and the objective lens 202 in order to perform autofocus. Movementis performed, for example, in parallel with an optical axis of theobjective lens 202. The objective lens actuator 203 may move the samplecontainer 100, the objective lens 202, or both.

The image forming lens 205 forms an image of a focus position of theobjective lens 202 on the camera 204. The camera 204 converts this imageinto an electrical signal to detect, for example, a spot image formed bya light beam.

The autofocus unit 206 is an optical unit that acquires a signal forperforming autofocus. The autofocus unit 206 includes a light beam lightsource 215 (to be described later with reference to FIG. 2 and the like)that emits a light beam toward an inner side of a bottom surface of thesample container 100.

The focus control unit 207 adjusts a focus position of the objectivelens 202 to the inner side of the bottom surface of the sample container100 (for example, an inner side of a bottom surface of the sample holder101) based on the signal from the autofocus unit 206.

The reflective surface identification unit 208 (determination unit)determines whether the light beam has been applied to a defect. Forexample, when autofocus processing is completed, it is determinedwhether the autofocus is normal or whether the light beam has beenapplied to the defect based on a spot image of the light beam on amicroscopic image of the camera 204.

The illuminator 209 is provided above the sample container 100. Theilluminator 209 is preferably a white or monochromatic LED. Theilluminator 209 is a transmissive illuminator in FIG. 1 , but may be areflective illuminator in which a sample is irradiated from directlybelow the objective lens 202 by a beam splitter or the like. Theilluminator 209 emits a light beam toward the inner side of the bottomsurface of the sample container 100.

A laser interferometry, a laser reflection method, an astigmatismmethod, a skew beam method, or the like can be used as the autofocusprinciple of the autofocus unit 206. The autofocus unit 206 irradiatesthe bottom surface of the sample holder 101 with light such as a laserthrough the objective lens 202, and detects reflected light from thebottom surface of the sample holder 101.

The focus control unit 207 determines whether a focus 201 of theobjective lens 202 output from the autofocus unit is too close to, toofar from, or in focus with a reflective surface from the objective lens202, and moves the objective lens actuator 203 according to adetermination result.

The XY stage 300 moves the sample container 100. In particular, it isdesirable to move in a plane perpendicular to the optical axis of theobjective lens 202.

The focus control unit 207, the reflective surface identification unit208, the XY stage 300, and the camera 204 are connected to the controlPC 400. Observation is performed by executing the autofocus processingaccording to a prescribed process and automatically acquiringmicroscopic images of cells in a sample liquid in the sample container100. PLC control, a control board, or the like may be used for thecontrol PC 400.

FIG. 2 shows the optical system in a state in which the autofocus iscompleted. A state is shown in which a bottom surface of one of theplurality of sample holders 101 of the sample container 100 is in focus.A defect 104 is present in a focus portion of the bottom surface(observation surface) of the sample holder 101.

The defect 104 hinders an appropriate operation of the autofocus, andincludes, for example, a crack in the sample container 100, a curve ofthe bottom surface of the sample container 100, a precipitate or animpurity in the sample liquid, unevenness in a pattern formed by thesample, and the like.

Illumination light having a uniform luminance distribution is emittedfrom the illuminator 209 for obtaining a microscopic image to a field ofview of a focus surface, and light from the focus surface is captured bythe objective lens 202.

The light in a visible region of the illuminator 209 is reflected by adichroic mirror 216, and the microscopic image of the focus surface isformed on a sensor of the camera 204 by the image forming lens 205.

When the astigmatism method is used for the autofocus principle, a lightbeam for autofocus is emitted from the light beam light source 215,becomes parallel light by the collimator lens 214, is emitted from belowthe dichroic mirror 216 through a beam splitter 213, and is emitted tothe bottom surface of the sample holder 101 through the objective lens202.

Most of the emitted light beam is reflected at a bottom portion of thesample holder 101, and a part of the light beam is scattered by thedefect 104. The reflected light beam is reflected by the dichroic mirror216 and the beam splitter 213.

The light beam reflected by the dichroic mirror 216 forms an image onthe sensor of the camera 204 in the same manner as the microscopicimage.

The light beam reflected by the beam splitter 213 forms an image on aphotodiode 210 (detector) through an image forming lens 211 and acylindrical lens 212. In this way, the photodiode 210 detects a spotimage formed by the light beam. In this example, both the camera 204 andthe photodiode 210 function as detectors that detect the spot image ofthe light beam, but only one of either the camera 204 and the photodiode210 may function as a detector in a modification.

2. Measurement Method by Observation Device First Embodiment

FIG. 3 shows a method for continuously capturing images of observationsurfaces 103 of the sample holders 101 in which a plurality of samplesolutions 102 are stored.

FIG. 3A shows a state at a time point when autofocus processing isstarted for one observation surface 103 of the plurality of sampleholders 101. The autofocus unit 206 irradiates the sample container 100with a light beam. The objective lens actuator 203 is driven to scanwhile moving the focus 201 upward. While scanning, luminance of thereflected light of the light emitted from the autofocus unit 206 iscontinuously detected. At this time, it is desirable that theilluminator 209 (light source for microscope imaging) is turned off.

As shown in FIG. 3B, an accurate focus operation may be performed by thefocus control unit 207 at a time point when the autofocus unit 206detects a peak in the luminance of the reflected light from theobservation surface 103. The accurate focus operation may be, forexample, a focus operation based on a known technique.

For example, the objective lens actuator 203 moves the objective lens202 in an optical axis direction of the light beam with respect to thesample container 100, thereby focusing a spot image.

In a state in which the spot image is in focus, the spot image of thelight beam is detected and captured by the camera 204. The reflectivesurface identification unit 208 determines whether the light beam hasbeen applied to a defect based on the captured spot image. A state inwhich the light beam has been emitted to a defect can be said to be astate in which autofocus is not normally performed, and a state in whichthe light beam has not been applied to a defect (that is, a state inwhich the light beam has been applied to a portion that is notdefective) can be said to be a state in which autofocus is normallyperformed.

In FIG. 3C, it is assumed that the light beam has been applied to adefect. In this case, the reflective surface identification unit 208determines that the light beam has been applied to the defect, and theXY stage 300 moves the sample container 100 with respect to theobjective lens 202 in a direction orthogonal to an optical axis of thelight beam according to a prescribed condition. Accordingly, the focus201 moves in an XY plane with respect to the observation surface 103.

In the present embodiment, the “prescribed condition” is notparticularly defined. That is, when it is determined that the light beamhas been applied to the defect, the XY stage 300 moves the samplecontainer 100 regardless of other conditions. As a modification, this“prescribed condition” may be defined as appropriate (an example will bedescribed in a third embodiment to be described later).

After the movement, the determination may be performed again. Forexample, the autofocus unit 206 irradiates the sample container 100 witha light beam, the camera 204 detects a spot image of the light beam, theobjective lens actuator 203 moves the objective lens 202 to focus thespot image, and the reflective surface identification unit 208determines whether the light beam has been applied to a defect based onthe spot image in focus. In this way, it is possible to avoid defectsand search for an appropriate portion.

When it is determined that the light beam has been applied to a defect,the processing may be ended. For example, prescribed error processingmay be performed, and the above processing may be performed on the nextimaging target (for example, the next sample holder 101).

The determination may be repeated a prescribed upper limit number oftimes until it is determined that the light beam has not been applied toa defect. For example, when it is determined that the light beam hasbeen repeatedly applied to a defect twice for the same imaging target,the processing may be ended.

When the reflective surface identification unit 208 determines that thelight beam has not been applied to a defect, a microscopic image of asample is captured using the illuminator 209. For example, as shown inFIG. 3D, the objective lens actuator 203 is fixed, a distance betweenthe objective lens 202 and the observation surface 103 is fixed, theobservation surface 103 is irradiated with illumination from theilluminator 209, and a microscopic image is acquired by the camera 204.

In this case, the observation device (for example, the reflectivesurface identification unit 208, and may be another component) may storea state of the moving mechanism at that time point (for example, aposition of the XY stage 300 and/or the objective lens actuator 203). Inthis way, a microscopic image can be acquired again at the same positionlater, which is suitable for a case of observing a change in the sampleover time, and the like.

FIG. 4 shows an operation flowchart of the above-described measurementmethod. This flowchart includes steps performed in the measurementmethod described above. When a change over time is observed (this isuseful when the sample is a biological sample, for example), a series ofoperations may be repeated at time intervals as desired.

FIG. 5 shows variations of spot images of a light beam. FIG. 5A is anexample of a spot image of the light beam in a case where the light beamhas not been applied to a defect (that is, in a case where autofocus isnormally operated). When a pattern of the spot image of the light beamdeviates from a pattern in FIG. 5A, it can be considered that the lightbeam has been applied to a defect.

Sizes of spot images in FIGS. 5B and 5C are different from that in FIG.5A. FIG. 5D shows a state in which an image of a defect is reflected ina spot image. FIG. 5E shows that a spot image is elliptical and asurface irradiated with the light beam is distorted. FIG. 5F shows thata spot image is translated and a surface irradiated with the light beamis inclined.

FIG. 6 shows an example of a criterion for determining whether a lightbeam has been applied to a defect. Luminance distribution of a spotimage is used in this example. As shown in FIG. 4 , such a determinationcriterion can be acquired at a start of the processing.

An image 501, as a spot image of a light beam, captured by the camera204 in FIG. 1 , is represented by data in which luminance values ofindividual pixels 502 are two-dimensionally arranged. An outer shape 505of the light beam in FIG. 6 is defined in advance corresponding to astate in which the light beam has not been applied to the defect asshown in FIG. 5A.

A pixel 503 inside the outer shape 505 and a pixel 504 outside the outershape 505 are defined in association with the outer shape 505. In thisexample, the inner pixel 503 is a pixel spaced inward from the outershape 505 by a prescribed distance or larger, and the outer pixel 504 isa pixel spaced outward from the outer shape 505 by a prescribed distanceor larger.

In a state in which the light beam has not been applied to the defect,it is considered that luminance of the inner pixel 503 is high andluminance of the outer pixel 504 is low. Therefore, when luminance ofall the inner pixels 503 is equal to or greater than a prescribed innerluminance threshold and luminance of all the outer pixels 504 is equalto or smaller than a prescribed outer luminance threshold, it can bedetermined that the light beam has not been applied to a defect.

Otherwise, that is, when luminance of any one of the inner pixels 503 issmaller than the inner luminance threshold or luminance of any one ofthe outer pixels 504 exceeds the outer luminance threshold, it can bedetermined that the light beam has been applied to a defect.

According to such a determination method, it is possible to detect astate in which the light beam has been applied to the defect as shown ineach of FIG. 5B to FIG. 5F.

Since the number of pixels of the spot image is smaller than the numberof pixels of the microscopic image, the determination can be efficientlyperformed.

As another example of the determination criterion, circularity of a spotimage may be used. For example, the outer shape 505 is acquired for thespot image, and the circularity of the spot image is evaluated. Anyknown method can be used for the evaluation of the circularity. When thecircularity is high, it is determined that the light beam has not beenapplied to a defect, and when the circularity is low, it is determinedthat the light beam has been applied to a defect.

FIG. 7 shows still another example of the determination criterion.Luminance distribution of a spot image is used in this example. In FIG.7A, a horizontal axis represents a position of a pixel by a distancefrom a center of the spot image, and a vertical axis representsluminance. In a state in which a light beam has been applied to adefect, it is considered that the luminance of the pixel at the centeris high, and the luminance decreases as the distance from the centerincreases. For this reason, in the example in FIG. 7A, an appropriaterange of the luminance of the pixel at the center is defined as a rangein which the luminance is high, and an appropriate range of theluminance of the pixel in a periphery is defined as a range in which theluminance is low.

When the luminance of all the pixels is within the appropriate range, itis determined that the light beam has not been applied to a defect.Otherwise, that is, when the luminance of any pixel is inappropriate(excessively large or excessively small), it is determined that thelight beam has been applied to a defect.

The example in FIG. 7B is an example in which an influence of adiffraction image in the spot image is further taken into consideration.A concentric diffraction image may appear superimposed on the spotimage, and an appropriate range changes stepwise in consideration of achange in luminance due to the diffraction image. In this way, it ispossible to perform the determination with higher accuracy by includingthe luminance of the concentric diffraction image in the spot image inthe luminance distribution.

These determination criteria can be used alone or in combination of twoor more. In addition, these determination criteria can each be expressedas a discriminant including an inequality sign.

The reflective surface identification unit 208 may determine whether alight beam has been applied to a defect based on a plurality of spotimages (for example, which will be described later with reference toFIG. 9 ) detected during execution of the autofocus processing. In thiscase, the determination may be performed individually for each spotimage, and the final determination may be performed based on theresults, or the determination may be performed once based on theluminance distribution or the circularity of each spot image. Accuracyof the determination is improved using the plurality of spot images inthis way.

As described above, according to a microscopic image capturing methodand a microscopic image capturing device according to the firstembodiment, the processing for performing autofocus on a portion havingno defect is improved.

For example, even on a focus surface having a defect, it is possible tomaintain high-accuracy autofocus by avoiding the defect. In addition, itis possible to cope with various defects, and it is possible to copewith unexpected random defects in some cases.

Second Embodiment

In the first embodiment, the criterion for determining whether the lightbeam has been applied to the defect is prepared in advance. In a secondembodiment, a more appropriate determination criterion can be acquiredby generating the determination criterion by machine learning.Hereinafter, the second embodiment will be described. Description of thesame points as those in the first embodiment may be omitted.

In the second embodiment, the reflective surface identification unit 208learns a criterion for determining whether a light beam has been appliedto a defect based on a plurality of data including a microscopic imageand a spot image of the light beam.

First, processing in a learning stage will be described. In the learningstage, first, teacher data is generated, and learning is performed usingthe generated teacher data.

FIG. 8 shows a flowchart of the processing in the learning stage of thedevice according to the second embodiment. The observation devicegenerates N teacher data (N is an integer greater than 1). N isdesirably 1000 or greater.

In the generation of the teacher data, the device acquires, for eachteacher datum, one microscopic image and one or more spot images of alight beam corresponding to the microscopic image. In a case where aplurality of spot images correspond to one microscopic image, the spotimages may be a series of images (process images) at different stages ofan autofocus operation (during a progress, at a completion time point,and the like of the autofocus operation). With such a configuration, itis possible to perform learning using spot images at various stages.

FIG. 9 shows examples of a progress image during execution of theautofocus processing. FIG. 9A shows progress images in a case whereautofocus is normally operated, that is, corresponds to teacher data ina case where a light beam has not been applied to a defect. FIG. 9A(1)is a spot image when the autofocus is completed (for example, when theobjective lens 202 is at an appropriate position). FIG. 9A(2) is a spotimage when the objective lens 202 is 10 µm frontward of an autofocuscompletion position. FIG. 9A(3) is a spot image when the objective lens202 is 20 µm frontward of the autofocus completion position.

FIG. 9B shows examples of the progress image in a case where a lightbeam has been applied to a defect, that is, corresponds to teacher datain the case where the light beam has been applied to the defect. Arelationship between each image and the autofocus processing in FIG. 9Bis the same as that in FIG. 9A. In this way, in the examples in FIG. 9 ,each teacher datum includes one microscopic image and a plurality ofspot images in which the position of the objective lens 202 is changed.

It is desirable to record each progress image until the autofocusprocessing is completed in association with a position of the objectivelens actuator 203 (for example, a relative position with respect to aposition where the autofocus processing is completed) or a time point(for example, a relative time point with respect to a time point whenthe autofocus processing is completed) .

FIG. 10 shows an example of the teacher data according to the secondembodiment configured as described above. Each teacher datum includes alabel (correct label) indicating whether a light beam has been appliedto a defect (“defect” in FIG. 10 ) or not (“normal” in FIG. 10 ). Amethod for determining the label is not limited, and the label may begiven by a person, or may be automatically determined by the reflectivesurface identification unit 208 or other components. In the presentembodiment, the reflective surface identification unit 208 automaticallydetermines the label as follows.

In the present embodiment, the reflective surface identification unit208 determines, for each teacher datum, a label of the teacher datumbased on a microscopic image included in the teacher datum. For example,the label is determined based on one or more of luminance distribution(that may include information on a luminance centroid), contrast,circularity, and the like of the microscopic image. In this way, it ispossible to appropriately determine the label for each teacher datum.

A case of using contrast will be described as a specific example. Thereflective surface identification unit 208 calculates the contrast ofthe microscopic image included in the teacher datum. The contrast can becalculated using, for example, a known technique. Next, the reflectivesurface identification unit 208 compares the calculated contrast with aprescribed threshold. This threshold can be specified in advance as areference value corresponding to a case where autofocus is normallyoperated, for example.

In a case where the calculated contrast is greater than the threshold,the teacher datum including the microscopic image is labeled as teacherdata in a case where a light beam has not been applied to a defect (thatis, autofocus is normally completed). In contrast, when the calculatedcontrast is equal to or smaller than the threshold, the teacher dataincluding the microscopic image is labeled that a light beam has beenapplied to a defect.

As described above, N teacher data are generated. After the label isassigned to the teacher datum, the microscopic image may be excludedfrom the teacher datum.

The reflective surface identification unit 208 performs learning usingthe generated teacher data. For example, each of the spot imagesincluded in the teacher datum is used as an input, a label is used as anoutput, and learning is performed such that correct output is performedfor the input (that is, such that a label output by the reflectivesurface identification unit 208 for the spot image matches the correctlabel associated with the spot image).

A specific configuration of a learning model and specific processing oflearning can be designed by a person skilled in the art as desired, andfor example, a support vector machine (SVM), a neural network, deeplearning, or the like can be used.

In this way, learning is performed, and a learned model is generated.The reflective surface identification unit 208 includes the generatedlearned model.

FIG. 11 is a flowchart of autofocus processing by the device accordingto the second embodiment. The processing according to the secondembodiment can be executed in the same manner as the first embodimentexcept that the determination criterion (discriminant) to be used isgenerated by machine learning as described above.

As described above, according to a microscopic image capturing methodand a microscopic image capturing device in the second embodiment, it ispossible to learn an appropriate determination criterion by machinelearning. Also in the second embodiment, the same effects as in thefirst embodiment can be attained.

Third Embodiment

A third embodiment partially changes the operation when it is determinedin the first embodiment or the second embodiment that the light beam hasbeen applied to the defect. Hereinafter, the third embodiment will bedescribed. Description of the same points as those in the firstembodiment or the second embodiment may be omitted.

As long as the accuracy of the determination by the reflective surfaceidentification unit 208 is not strictly 100%, it may be erroneouslydetermined that the light beam has been applied to the defect even ifthe autofocus is actually completed normally. In such a case, movementof the sample container 100 by the XY stage 300 is not actuallynecessary, but such movement is performed in the first embodiment. Thethird embodiment reduces such unnecessary movement.

FIG. 12 is a flowchart of autofocus processing of the device accordingto the third embodiment. When it is determined that a light beam hasbeen applied to a defect, the reflective surface identification unit 208further performs re-determination based on a microscopic image.

Specifically, in the re-determination processing, the reflective surfaceidentification unit 208 captures a microscopic image of a sample usingthe illuminator 209 (light source for microscope imaging). Then, thereflective surface identification unit 208 determines whether an imageof the defect has been captured based on the microscopic image.

In this determination, for example, it is possible to determine whetherthe image of the defect has been captured based on luminancedistribution (that may include information on a luminance centroid),contrast, or circularity of the microscopic image. A determinationcriterion in this case can be determined in the same manner as thecriterion used for determining the label of the teacher datum in thesecond embodiment. By using such a determination criterion, it ispossible to appropriately determine whether the image of the defect hasbeen captured.

When it is determined that the image of the defect is not captured (thatis, image capturing is normally completed), the observation device endsimage capturing processing on the sample (for example, turns off theilluminator 209). In this case, the microscopic image of the sample maybe captured again.

In contrast, when it is determined that the image of the defect has beencaptured, the moving mechanism (for example, the XY stage 300 and/or theobjective lens actuator 203) moves the sample container 100 with respectto the objective lens 202 in a direction orthogonal to an optical axisof the light beam. This processing corresponds to the processing in thecase where it is determined in the first embodiment that the light beamhas been applied to the defect. Thereafter, the autofocus processing isexecuted again.

As described above, according to a microscopic image capturing methodand a microscopic image capturing device in the third embodiment, in acase where it is erroneously determined that the light beam has beenapplied to the defect even though the autofocus is actually operatednormally, unnecessary re-autofocus processing can be omitted. Also inthe third embodiment, the same effects as in the first embodiment can beattained.

Reference Signs List

-   100 sample container (container)-   101 sample holder-   102 sample solution-   103 observation surface-   104 defect-   200 imaging unit-   201 focus-   202 objective lens-   203 objective lens actuator (moving mechanism)-   204 camera (detector)-   205, 211 image forming lens-   206 autofocus unit-   207 focus control unit-   208 reflective surface identification unit (determination unit)-   209 illuminator (light source for microscope imaging)-   210 photodiode (detector)-   212 cylindrical lens-   213 beam splitter-   214 collimator lens-   215 light beam light source-   216 dichroic mirror-   300 XY stage (moving mechanism)-   501 image-   502, 503, 504 pixel-   505 outer shape

1. A microscopic image capturing method for capturing a microscopicimage using a microscopic image capturing device, wherein themicroscopic image is a microscopic image of a cell or a particle as asample in contact with an inner side of a bottom surface of a container,the microscopic image capturing device includes a transparent containerconfigured to accommodate a sample, a light source for microscopeimaging, a light beam light source configured to emit a light beamtoward the inner side of the bottom surface of the container, anobjective lens used to form a spot image of the light beam reflected bythe sample or the container, a detector configured to detect the formedspot image, and a moving mechanism configured to relatively move thecontainer and the objective lens, and in the method, the microscopicimage capturing device further includes a determination unit configuredto determine whether the light beam has been applied to a defect basedon the spot image, the microscopic image capturing method comprising: astep of emitting the light beam from the light beam light source; a stepof detecting the spot image by the detector; a step of focusing the spotimage by the moving mechanism moving the objective lens in an opticalaxis direction of the light beam with respect to the container; a stepof determining by the determination unit whether the light beam has beenapplied to a defect based on the spot image, a step a) of, when it isdetermined that the light beam has been applied to the defect, moving bythe moving mechanism the container with respect to the objective lens ina direction orthogonal to an optical axis of the light beam according toa prescribed condition; and a step b) of, when it is determined that thelight beam has not been applied to the defect, capturing a microscopicimage of the sample using the light source for microscope imaging. 2.The microscopic image capturing method according to claim 1, furthercomprising: when it is determined that the light beam has been appliedto the defect in a), after the step of moving by the moving mechanismthe container with respect to the objective lens in the directionorthogonal to the optical axis of the light beam, a step of emitting thelight beam from the light beam light source; a step of detecting thespot image by the detector; a step of focusing the spot image by themoving mechanism moving the objective lens in the optical axis directionof the light beam with respect to the container; and a step ofdetermining by the determination unit whether the light beam has beenapplied to a defect based on the spot image.
 3. The microscopic imagecapturing method according to claim 1, wherein the determination unitdetermines whether the light beam has been applied to the defect basedon luminance distribution or circularity of the spot image.
 4. Themicroscopic image capturing method according to claim 3, wherein theluminance distribution is luminance distribution including luminance ofa concentric diffraction image in the spot image.
 5. The microscopicimage capturing method according to claim 1, wherein the determinationunit learns a criterion for determining whether the light beam has beenapplied to the defect based on a plurality of teacher data including themicroscopic image and the spot image.
 6. The microscopic image capturingmethod according to claim 5, wherein each teacher datum includes onemicroscopic image and a plurality of spot images in which a position ofthe objective lens is changed.
 7. The microscopic image capturing methodaccording to claim 5, wherein each teacher datum includes a labelindicating whether the light beam has been applied to the defect, andthe determination unit determines, for each teacher datum, the label ofthe teacher datum based on a microscopic image included in the teacherdatum.
 8. The microscopic image capturing method according to claim 7,wherein the determination unit determines the label based on luminancedistribution, contrast, or circularity of the microscopic image.
 9. Themicroscopic image capturing method according to claim 1, wherein when itis determined that the light beam has not been applied to the defect inb), the microscopic image capturing device stores a state of the movingmechanism at that time point.
 10. The microscopic image capturing methodaccording to claim 1, wherein the determination unit determines whetherthe light beam has been applied to the defect based on a plurality ofspot images detected during execution of autofocus processing.
 11. Themicroscopic image capturing method according to claim 1, furthercomprising: when it is determined that the light beam has been appliedto the defect in a), a step of capturing a microscopic image of thesample using the light source for microscope imaging; a step ofdetermining by the determination unit whether an image of the defect hasbeen captured based on the microscopic image; and a step of, when it isdetermined that the image of the defect has been captured, moving by themoving mechanism the container with respect to the objective lens in thedirection orthogonal to the optical axis of the light beam.
 12. Themicroscopic image capturing method according to claim 11, wherein thedetermination unit determines whether the image of the defect has beencaptured based on luminance distribution, contrast, or circularity ofthe microscopic image.
 13. A microscope image imaging device configuredto perform the method according to claim 1.